Preprint typeset using lAT^^X style cmulateapj v. 04/03/99 



A NICMOS SEARCH FOR HIGH REDSHIFT ELLIPTICAL GALAXY CANDIDATES ^ 

TOMMASO Treu^ and Massimo Stiavelli ^'^ 

Space Telescope Science Institute, 3700 San Martin Dr., 21218 MD, U.S.A.; treu@stsci.edu, 

mstiavel@stsci.edu 



< 



> 
oo 

o 

0^ 

ON ' 

I 

O 



ABSTRACT 

We have collected optical follow-up observations from the ground and the Hubble Space Telescope 
(HST) for 13.74 □' of archival images taken with HST Near Infrared Camera and Multi Object Spectro- 
graph (NICMOS). We use two criteria to select E/SO galaxy candidates at z ^ 1 and z ^ 1.5 (HizECs) 
based on colors and infrared morphology. The observed surface density of HizECs is significantly smaller 
than what is predicted assuming the local luminosity function (LF) of E/SO, constant comoving density, 
and pure passive evolution with a high redshift of formation of the stellar populations (z ^ 5). On the 
other hand, assuming a low redshift (z <^ 2) of formation, not enough HizECs are predicted. The data 
are very well matched assuming that a substantial fraction of present-day E/SOs (10-66%, depending on 
cosmology and on the local LF of E/SOs) formed at z = 3 or higher and then evolved passively. The rest 
of the current E/SO population may have formed at lower redshift, or may have been rejected by the 
selection criteria because of interactions or more recent episodes of star-formation (disturbed morphology 
and/or bluer colors). 

Subject headings: galaxies: elliptical and lenticular, cD — evolution — formation — ; cosmology: 
observations; infrared: galaxies 



1. INTRODUCTION 

The formation of elliptical galaxies is still not under- 
stood. In a schematic view, all the different models can be 
summarized within two simple paradigms: i) the mono- 
lithic collapse, in which ellipticals form at high redshift in 
short bursts of star formation; and ii) the merging sce- 
nario in which they form at lower redshift, by merger of 
disk galaxies. 

In recent years, many observational programs have 
started to provide useful constraints on theoretical mod- 
els. Studies such as MORPHS, a Hubble Space Tele- 
scope (HST) sur vey of intermedi ate redshift clusters (up 
to z « 0.5; e.g. Ellis et al. 1997| ), or the investigation of 
the Fundamenta l Plane of ellipticals at i ntermediate red- 
shift in clusters (van Dokkum et al. 199S) and in the field 



and morphological information is the best way, short of 
spectroscopic identification, to distinguish between high 
redshift old stellar populations and reddened dusty star 



(Treu et al. 199£), indicate that the bulk of star forma- 
tion occurred at z ^ 1. However, low redshift (z 1) 
diagnostics are mostly sensitive to the epoch when stars 
were formed rather than to that when galaxies were as- 
sembled (see however Lilly et al. 1995; Kauffmann, Char- 
lot & White 1996). Unfortunately, it is in their predic- 
tions on the latter that galaxy formation models differ the 
most. Thus, the most direct and uncontroversial way to 
break the degeneracy is to push the investigation to higher 
and higher redshift, trying to put limits on the number of 
already assembled (proto-) ellipticals. The Near Infrared 
Camera and Multi Object Spectrograph (NICMOS) par- 
allel images can provide a decisive breakthrough onto this 
issue. In fact, high redshift ellipticals appear extremely re d 
in their optical-infrared colors (e.g. ^pinrad et al. 19"97 ), 



bursts (e.g., Graham & Dey 1996; Stiavelli et al. 1999 
hereafter S99). 

By using the NICMOS parallel images combined with 
deep optical images, both from ground based telescopes 
and with the Wide Field and Planetary Camera 2 
(WFPC2) on HST, we have undertaken a survey (13.74 □') 
to determine the number of ellipticals already assem- 
bled at z ^ 1. The aims of the survey, the selection 
criteria and the sample are described in Sec. |^. The 
building of the catalog is briefly discussed in Sec. |[ 
The results and conclusions are discussed in Sec. IJ. A 
more detailed description of the data reduction, the com- 
plete catalog of the selected sources, together with im- 
ages, surface photometry and photometric redshifts, will 
be given elsewhere (Paper II, in pre paration. See also 



http: / /icarus.stsci.edu/'-^treu/HizECs ) 



AB magnitudes ( |Oke, 1974 ) are used throughout the 
paper. The HST filters are indicated as: Hi6=F160W and 
Jii=F110W (NICMOS); l8i4=F814W and V606=F606W 
(WFPC2). Ho is assumed to be 100 h km/s/Mpc. The 
present value of the matter density and the cosmological 
constant are indicated by O and J^a, respectively. 

2. THE SURVEY 

Due to the effect of redshift and the lack of UV emission 
in old stellar populations, ellipticals at z=l-2 are relatively 
bright in the IR, but they are faint in the optical. For ex- 



ample a single burst model of solar metallicity (Bruzual & 
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Chariot 1993; GISSEL 96 version; hereafter BC96) with 
Salpeter IMF and cutoffs at 0.1 and 100 Mq, normalized 
to a luminosity of My=-21.0 at 15 Gyrs (roughly M^,), at 
2 Gyrs and z=1.5, has Hi6o=21, Jiio=22, I8i4=23.7, R = 
25.3, Vgoe — 26, assuming H. = 0.3 and J^a = 0.7. For this 
reason, high redshift ellipticals can constitute a significant 
fraction of the so-called Extremely Red Objects (EROs). 
Unfortunately, even the best color-based classification suf- 
fers from the degeneracy between old stellar populations 
and dusty starbursts. High-resolution imaging holds the 
key to classify these objects properly since many EROs 
are so faint that their continuum cannot be detected spec- 
troscopically, even with 8-10 meter telescopes. For exam- 
ple, the nature of the two best known EROs with spec- 
troscopic information is fully revealed by thei r morphol- 
ogy: i) HRIO, a dusty s tarburst at z=lAA (Dey et al 
19991, [Cimatti ct al. 19981) is distorted in the HST images 



while a) LBS53W091 is smooth and shows a perfectly reg- 
ular r^/^ light profile (Dunlop 1998). From the statistics 
of EROs we know that the density of high redshift ellip- 
ticals (z <; 1) must be of the order of l/D', the precise 
value depending on the definition and the magnitude limit 
(e.g. ^Thompson fc Beckwith 199^ ; |Cohcn et al. 1999| ; [Yan. 
19990 

For all these reasons, the essential ingredients for a suc- 
cessful search for high redshift ellipticals are: 1) High res- 
olution IR imaging. The morphology is best studied in the 
IR, where sources are brighter and the surface brightness 
is a better tracer of the mass distribution and is less af- 
fected than the optical by, e.g., a dwarf galaxy companion 
or an otherwise negligible amount of star formation. 2) 
Medium deep IR imaging, as the galaxies we are looking 
for are expected to be in the range Hi6=20-24. Deeper 
images would probe a wider range of the luminosity func- 
tion, but it would be difficult to measure the flux of the 
optical counterparts. 3) The largest possible area. 

2.1. Selection Criteria 

In order to count high redshift elliptical candidates 
(HizECs) properly we need a definition excluding as much 
as possible galactic stars and dusty star-forming objects. 
We require the object to be: i) clearly resolved, i.e. 
FWHM ^ 0'.'3, hence having a scale length ^ 3 kpc, de- 
pending on the redshift and the cosmological parameters; 
a) to appear roundish and regular, with no spikes, tails 
or knots; Hi) to have colors consistent with a BC96 model 
of old stellar populations (^ 1 Gyr) at z ^ 1. In or- 
der to quantify the third criterion, we have computed the 
observed colors of a range of models as a function of red- 
shift. For a given cosmology and metallicity we computed 
the colors of a passively evolving stellar population formed 
at redshift z/=10, and of a 1 Gyr old single burst stellar 
population. These two cases represent the red and blue 
extrema of the objects we want to select and therefore 
bracket the possible cases. Based on the computed colors 
we define a broad and a narrow selection criteria. The 
broader criterion, tuned to collect all the elliptical candi- 
dates at 2 ^ 1, is: I8i4-Hi6 > 1.8, 1-Hig >1.9, R-Hie >3.2, 
and Vgoe-Hie >3.8. The more stringent criterion, tuned 
to z > 1.5 is defined by Isu-Hie > 2.7, I-Hig > 2.8, R- 
Hi6 > 4.0, Vgoe-Hie > 4.5. As the Jn-Hig color is much 

^Only in field 1509-1126 the probability goes down to 77 % at H= 



smaller, and therefore more affected by photometric (of 
order of 0.2-0.3 mags) and modeling errors, we do not in- 
clude it into the color selection criterion, but we shall use 
it, when available, as a consistency check. 

Extremely high redshift ga laxies can app ear very red 
when Lya Forest absorption ( Madau, 1995| ) starts to af- 
fect the Vgoe and R filters. Nevertheless, the colors of ex- 
tremely high redshift galaxies can easily be distinguished 
from those of an old stellar population at z=l-2 because 
the cosmic transmission falls bluewards of 1215 A much 
more sharply than the expected HizECs Spectral Energy 
Distribution ( [Madau et al. 19^ ). Furthermore, galaxies 
at z > 6 would need to be enormously bright in order to be 
detectable at H~ 24 (Chen, Lanzetta & Pascarelle, 1999; 



Weymann et al. 1998). Therefore we assume the contami- 
nation to our sample of HizECs by extremely high redshift 
galaxies to be negligible. 

2.2. The sample 

In order to survey the largest possible area we collected 
data from different sources: 1) The pointings of the HST 
NICMOS parallel data taken with NIC3 through F160W 
were cross-correlated with the entire WFPC2 and STIS 
archive, looking for intentional or serendipitous overlaps 
with WFPC2 red, or STIS clear images. Only overlaps 
with I8i4 and Veoe were found. We selected the fields with 
exposure time greater than 1000 s. Fields with gal actic lat- 
itude |b| < lOdeg, or E(B-V)>0.15 (according to Bchlegej 
et al. 1998) were discarded to reduce intervening redden- 
ing or foreground star contamination. Fields with HII fila- 
ments or with the primary target extended enough to enter 
the parallel field of view were discarded as well. 2) Deep 
optical R band imaging of two NICMOS parallel fields was 
obtained at the ESO-NTT using SUSI2 under superb see- 
ing (0'.'5-0'.'6 measured on the combined image). The 3(j 
limiting magnitude over a ID" area is R=26.8. 3) The op- 
tical counterparts for the HDFS NICMOS flanking fields 
made available by Teplitz et al. (1999). We also used the 
R and I band images of Treu et al. 1998 (hereafter T98). 
4) The NICMOS catalog of part of the HDF (Thompson 
et al. 1999) was cross-correlated with the HDF WFPC2 
catalog (Wilhams et al. 1996). The HDFS NICMOS field 
(Fruchter et al. 1999, in preparation) was also considered 
in the survey, together with the VLT Science Verification 
optical data (Renzini 1998). 

The 23 fields are listed in Tab |l] together with the expo- 
sure times in different bands, and the effective area cov- 
ered. The latter was computed as the intersection of the 
IR and optical fields of view. We ran extensive simulations 
(> 10,000 per field) in order to measure the probability of 
detection of a compact elliptical-like source as a function 
of magnitude, and to correct the raw counts for complete- 
ness. At our magnitude limits the completeness was always 
higher than 90 



3. BUILDING THE CATALOG 

The pedestal (e.g., T98) was subtracted from the cal- 
ibrated data by using a dedicated software. The images 
were otherwise reduced in the standard way. The pre- 
l iminary Hig catalog wa s obtained by running SExtractor 
(Berlin & Arnouts 1996) on the coadded images. The de- 

:23.5 due to the presence of a bright star. 
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tected sources were inspected on the single frames to avoid 
defects, and the magnitudes were checked against aper- 
ture photometry. We considered as resolved all the sources 
with FWHM as measured with imexamine greater than 
the FHWM of the stars on the coadded frames (resolution 
quality 1). When the signal to noise was high enough, the 
dimensions were checked on the single frames. In fields 
with no stars, we considered as resolved all objects with 
a FWHM greater than 1.5 pixels (resolution quality 2). 
When WFPC2 images were available and the objects were 
detected in optical, we required them to be resolved also 
in the optical image. Resolution quality 3 was assigned to 
objects with FWHM similar to the stars or of the order 
of 1.5 pixels if no stars were present. Resolution qual- 
ity 4 was assigned to objects more compact than stars or 
smaller than 1.5 pixels, if no star were present. Only ob- 
jects with resolution quality 1 and 2 were considered in 
the final sample. 

As the main aim of this survey is to provide a morpho- 
logically selected sample of HizECs, we limited the Hig 
band catalog to the relatively bright sources. The limiting 
magnitudes listed in Tab. 1 were obtained by cutting each 
luminosity sorted catalog at the level where all the sources 
were detected with a magerr_best SExtractor parameter 
smaller than 0.15 (and then rounded to the next bright- 
est half magnitude). Unfortunately, the HDFS NICMOS 
flanking fields have a relatively shallow optical counter- 
part and therefore it was pointless for our purposes to go 
fainter then Hi6=23.5. The limit Hi6=24.5 was used for 
the HDF and the HDFS, where deeper optical images are 
available. 

After cross-identification of the brightest sources, the 
Hi6 images were aligned with the other passbands and 
aperture photometry was performed for all the resolved 
objects. We used large apertures in order to measure to- 
tal magnitudes. As a double check, SExtractor was run on 
the optical images. The two photometries were consistent 
within the errors for all the objects in the final catalog. 

The objects not detected in the optical were assigned 
an upper limit to the luminosity corresponding to the 3(t 
background fluctuation on a fixed aperture of 0.25 □" for 
the HST images, and two times the FWHM for the CTIO 
and NTT images. As the CTIO images were taken un- 
der different seeing conditions, with different pointings and 
cameras, we did not combine them but instead we com- 
puted the limiting magnitude from the combined probabil- 
ity of non-detection. The combined limiting magnitudes 
are 1=25.7 and R=26.45. When only the images by Teplitz 
et al. (1999) were available the 3 a limiting magnitudes 
were measured to be 1=25.6 and R=26.05. 

The colors were finally corrected for galactic extinc- 
tion using the coefficients and the E(B-V) values given 
by Schlegel, Finkbeiner & Davis (1998). 

All the resolved objects in the catalog that satisfied the 
color selection criteria were morphologically classified in- 
dependently by the authors, dividing them into three mor- 
phological bins: E/SO, S, Ir. Only the objects for which 
we agreed on the classification were assigned a morpholog- 
ical type. 

4. RESULTS AND CONCLUSIONS 

The observed integrated surface density of HizECs as 
a function of magnitude is shown in Fig. |l| for the low 



(hexagons) and high (triangles) redshift selection criteria. 
It is worth stressing that this measurement was performed 
in random fields whereas many of the known EROs or high 
redshift ellipticals are radio selected, or found in the vicin- 
ity of QSOs or AGN, so that their inferred surface density 
may be biased. 

Fig. H also shows the predictions of a class of models. 
These are similar to the ones computed by Zepf (1997), 
and assume a constant comoving density, passive evolu- 
tion after a single burst of star formation at z = Zf, and a 
Schechter local luminosity function (LF). The parameters 
for the local LF were taken from Gardner et al. (1997) scal- 
ing th e total normaliz ation factor to the fraction of E/SO 
(32%; Gardner, 1998). The predictions of the models are 



shown as solid (high redshift) and dotted (low redshift) 
lines for different cosmologies and different redshift of for- 
mation. In each case we scaled the normalization factor in 
order to match the observed counts. We confirm previous 
resuhs (Cowie et al. 1994; Zepf 1997; Barger et al. 1999; 
Kodama, Bower & Bell, 1999; see however Benitez et al. 
1999) that fewer objects are observed than predicted by a 
passively evolving model of galaxies formed at very high 
redshift (5 or 10 depending on cosmology). In addition it 
can be noticed that: 1) Models with a very low redshift 
of formation for the stellar populations of E/SOs (z ^ 2, 
cases (a) and (b) in Fig. 1) do not account for the ob- 
served population of HizECs, as no object red enough is 
predicted. 2) Models where a significant fraction of the 
current E/SO population was already assembled at z—l~2 
and the stellar population was formed at higher redshift (5 
or 3) describe the observed counts well. The exact fraction 
of objects depends to some extent on the normalization of 
the local LF, but the results are qualitatively unchanged 
if different E/SO LFs are used. For example, using the pa- 
rameters from Marzke et al. (1998), about 25% of current 
E/SOs are needed to fit our data for case (e). 3) If we halve 
the local LF, assuming that only Es form in short bursts 
of star formation at high redshift, the fraction of progeni- 
tors already assembled at high redshift doubles: the data 
are consistent with a scenario where a large fraction of Es 
form at high redshift and evolve passively, while the SOs 
form later or undergo late episodes of star formation or 
interactions. Surface photometry can help addressing this 
issue. 4) It is possible to break the degeneracy between 
redshift of formation and fraction of population by means 
of two selection criteria sensitive to slightly different red- 
shift intervals. For example model (d) can be made to 
match the broad criterion number counts, but does not 
predict any object red enough to satisfy the high redshift 
criterion. The same qualitative behavior can be noticed in 
model (c) even though the errors are still too large to rule 
out this particular model. 

It should be noticed that our measurement of HizECs 
number density may be considered as a lower limit on the 
actual number of objects because: i) very compact, not 
resolved, E/SOs drop out of the sample as they are not 
resolved; ii) E/SOs may be rejected by their morphology 
if, e.g., they are disturbed by a dwarf companion or by 
an interaction. In addition, single burst models produce 
the reddest possible colors for a given Zf, cosmology, and 
metallicity. Therefore also the color selection criteria are 
tuned to produce a lower limit. Unfortunately, the resid- 



4 



High redshift ellipticals 



ual possibility of contamination from regular-looking dusty 
starbursts prevents us from pushing the lower limit inter- 
pretation of our measurement. A spectroscopic follow up 
of the brightest candidates is needed to asses the magni- 
tude of this contamination, if at all present. 

Taking our result at face value, a large fraction of the 
zero redshift E/SO population is not detected as HizECs 
at z^ 1. Possible interpretations are that they may have 
been assembled at lower redshift, may be interacting, or 
they may be bluer than required by our color selection 
criteria due to a more complicated star formation history 
with delayed star formation (e.g., Jimenez et al. 1999). We 
plan to test the prediction of alternate models in paper II. 

We thank R. P. van der Marel who developed the 
Pedestal Extraction Software; O. Hainaut who took the 
NTT images; P. T. de Zeeuw for a careful reading of the 
manuscript; M. CaroUo for useful discussion. We used 
NED, ADS, and the LANL e-print archive. This research 
was partially funded by STScI DDRF grant 82216. 
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Fig. 1. — Integrated number density of HizECS for the low (hexagons) and high (triangles) redshift selection criteria. Simple model 
predictions, assuming the LF from Gardner et al. (1998), constant comoving density and passive evolution are overplotted as solid (high 
redshift) and dashed (low redshift) lines (see text for details). 
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Table 1 

Observed fields and exposure times 



a 


S 


tl60 


t814/I 


teoe/R 


mi60 


Area 


00 17 


+15 43 


512 




2200" 


22 


0.6778 


07 38 


+05 07 


1536 




1800" 


23 


0.7019 


09 20 


+69 09 


512 




1500" 


22 


0.6300 


10 56 


-03 37 


1024 


6500° 


6500" 


23 


0.7212 


13 11 


-01 20 


1024 


2300" 


1800" 


22.5 


0.1713 


13 38 


+70 12 


256 


1700" 


1160" 


21.5 


0.4946 


13 38 


+70 13 


256 


1700" 


1160" 


21.5 


0.1330 


14 55 


-33 33 


2048 




3600*' 


22.5 


0.7281 


15 09 


-11 26 


3071 




3600*" 


23.5 


0.6970 


21 50 


+28 45 


512 


8100" 


7100" 


22.5 


0.6806 


22 08 


-19 47 


1024 




1000" 


22.5 


0.2546 


22 32 


-60 38 


128441 


10158= 


7200= 


24.5 


0.9569 


22 33 


-60 40 


5376 


7900^* 


7800'' 


23.5 


0.6139x7 


22 33 


-60 40 


5376 


5100<^ 


4800'' 


23.5 


0.6110x2 


22 32 


-60 43 


5824 


5100<^ 


4800'' 


23.5 


0.7084= 


12 36 


+62 12 


131712 


123600 


109050 


24.5 


0.6601 



Note. — The coordinates (hh and mm) axe J2000, exposure times are 
in seconds, areas are in □'. The column labeled mieo lists the limit- 
ing magnitude adopted for the Hie catalog. The flanking fields with the 
same exposure times have been grouped together, a) I8i4 and Veoe HST 
WFPC2 data; b) ESO NTT SUSI2 c) ESO VLT-UTl Science Verifica- 
tion. Also VLT-Test Camera U B and V, STIS CLEAR, and CTIO I, R, 
V, U, arc available, d) R and I taken at the CTIO 4m Blanco Telescope; 
e) Total area covered by two fields partially overlapping. One of the 
fields has ti6o=7296. 
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n -0, h = 0.5 n-0.3, n -0.7, h = 0.65 




